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PRESENTED ARE GUIDELINES FOR THE EVALUATION OF PROGRAMS 
FUNDED IN THE PENNSYLVANIA SCHOOLS. UNDER TITLE I OF THE 
ELEMENTARY AND SECONDARY EDUCATION ACT. SPECIFICALLY 
DISCUSSED ARE GUIDELINES FOR THE LOCAL EDUCATIONAL AGENCIES 
TO USE IN REPORTING THE MEASURING INSTRUMENTS AND EVALUATION 
TECHNIQUES USED IN THE EVALUATIONS AND THE ADMINISTRATIVE 
STRUCTURE AND PERSONNEL INVOLVED. THERE ARE ALSO INSTRUCTIONS 
ON HOW TO PRESENT CONCLUSIONS. INCLUDED ARE COPIES OF THREE 
EVALUATION FORMS TO BE FILLED OUT, WITH DIRECTIONS ON THEIR 
USE. THE DEPARTMENT OF PUBLIC INSTRUCTION OF PENNSYLVANIA HAS 
ESTABLISHED A CENTRAL OFFICE TO PROVIDE FOR COMPLIANCE WITH 
THE TERMS OF THE ACT. (NH) 
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ESSENTIALS OF EVALUATION OF EDUCATIONAL OUTCOMES 

1. Evaluation is the analysis and interpretation of measurements 
of educational outcomes. 

2. Evaluation presupposes a determination and statement of educa- 
tional objectives or goals based on the educational needs of 

a selected group of children.' 

3. Evaluation also implies the determination of personnel, materials 
and techniques to be employed for the attainment of the stated 
objectives. 

4. The entire educative process, therefore, consists of stating 
objectives, determining methods, and measuring the outcomes. 

5. Evaluation, in the final analysis, is a measurement of the degree _ 
to which the objectives of an educational enterprise have been 
attained by means of the methods employed. 

6. Evaluation may be in terms of direct evidence (objective measures) 
or indirect evidence (subjective measures). 

7. Direot or objective measures consist of tests of determined 
reliab ili ty (the consistency with which a test makes measurements) 
and validity (the extent to which a test measures what it is de- 
signed to measure), and for which standards or norms have been 
established. 

8. In dir ect or subjective measures consist of judgments based on 
observation, and the results obtained from using such devices as 
questionnaires, opinionnaires , and rating scales. 



I 




- 1 - 




f 



EVALUATION OF ESEA-TITLE I PROJECTS 

Introduction : Title II of Public Law 81-87^ as amended by Public Law 

89-10, Title I, stipulates that all activities conducted in accordance with 
and financed by this legislation shall be evaluated. State Educational 
Agencies are charged under this Law- to direct and implement the evaluation 
of all Title I projects conducted within their jurisdictions. State Edu- 
cational Agencies are also required by this Law to report annually an 
evaluation summary of all such projects to the United States Office of 
Education. 

Provision has been made in Pennsylvania for compliance with this man- 
date through the Education Evaluation Section of the Office of School and 
Program Evaluation in the Department of Public Instruction. Forms, explana- 
tions and directions on evaluation of ESEA-Title I projects have been pre- 
pared and are being disseminated by this office. Reports will be collected 
and results tabulated and analyzed in this office. Also, the reports on 
evaluation for the United States Office of Education will be prepared here 
and transmitted as required. 

The Local Educational Agency which conducts one or more ESEA-Title I 
projects is charged with the responsibility of evaluating the educational 
outcomes achieved thereby. Every effort and care should be exercised to 
make these evaluations just as accurate and meaningful as. it is possible 
for them to be. Forms and directions have bean prepared in the Office of 
Evaluation to enable Local Educational Agencies to make and report accurate 
and comprehensive evaluations. 
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Measuring Instruments : The specific objective instruments which have 

been or will be used both as pre-tests and later as measures of outcomes 
should be reported, 

1. Name the standardized tests, the forms used, the date 
of publication, and indicate whether national or local 
norms are employed. 

2. Describe any locally devised tests which are used and 
report information on the reliability and validity of 
each such test, 

3. Describe any non-standard instruments such as attitude 
scales, level of aspiration measures, occupational 
preference scales, cognitive styles and problem solving 
measures, etc., which are used. 

Objective instruments have not yet been devised for measuring many 
desirable educational outcomes. In these cases it is necessary to rely on 
the judgment of competent personnel. 

Competence to make valid judgments may be expected to result from 
maturity, experience and training. The opinions of e:qperienced teachers, 
supervisors, counselors, and subject area specialists should have acceptable 
validity. The statements of parents may or may not be valid indications of 
educational achievement. 

In those cases where subjective judgment is used as a measure of edu- 
cational outcomes, the competence of the person or persons making such 
statements should be assessed and reported. 



Measuring Techniques : It is necessary to indicate how much and what 

kinds of changes in the performance in the target population occurred as 
a result of the activities of an EEIEA-Title I project. The nature and 
degree of change should be stated in terms which are indicative of a suc- 
cessful expenditure of Title I funds. . It is also necessary to indicate 
how the changes are related to the project and its various activities. 

The changes expected and achieved in the target population should 
reflect the judgment of the Local Educational Agency (LEA) as to what can 
be considered a successful expenditure of Title I funds in terms of per- 
centile scores or some other method of measuring changes in performance. 
These changes should include, where appropriates 

1. Changes in means of distributions of scores of children 
tested. 

2. Changes in cut-off scores for quartiles (Q^ for lowest 
quartile and for highest quartile) for children tested. 

An accurate evaluation technique is to make comparisons between the 
achievement of the target population and control groups. These comparisons 
may be based on the achievement of participating and non-participating groups 
or comparable populations of previous years, or other normative populations 
to whom the target population may be compared. 

The control groups as well as the target populations can be expected 
to show changes in scores over a period of time. It is necessary to indi- 
cate the extent the target population changes its standing in comparison 
with the control group which was selected. Such a change in standing may 
be expected to occur sooner as a result of some activities than as a result 
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of others. The time when the major effects of the activities included in 
the project become clear should be indicated. 

A basic technique for measuring the outcomes of an ESEA-Title I project 
is: first, to determine the needs of the educationally deprived group by 

means of tests and measures administered before the inauguration of the 
project; and second, to repeat the same measures during or at the completion 
of the project to determine the kind and amount of growth which has resulted 
from the project. 

Administrative Structure and Personnel Responsible for Evaluation : 

The measurement of educational outcomes requires special competence. This 
competence is attained through training in educational measurements. To be 
assured that the evaluation of ESEA-Title I projects will be accurate, it 
is advisable that it be done by persons who have had such training. 

The title and position of the person responsible for evaluation of a 
project should be reported. In the absence of such a person on the staff 
of the LEA., it may be desirable to call in consultants or contract with 
some available agency to perform the evaluation service. 

Whether evaluation is conducted by staff members or by an outside agency 
the qualifications of the persons responsible for making the measurements 



should be indicated 



Conclusions ; In addition to measuring the achievement of objectives, 
evaluation serves to establish the validity of the objectives. It also 
measures the appropriateness and the effectiveness of the methods employed 
and the materials used. 

The results of an evaluation should constitute the basis for determin- 
ing changes in an educational program which is a continuous enterprise. 

This is to say that the results of evaluation should serve a larger purpose 
than the accumulation of data to be recorded or reported on a form or placed 
in a file. 

If evaluation is to serve this ultimate purpose of educational dir- 
ection, the facts and data which are accumulated must be analyzed and inter- 
preted. Analysis and interpretation of data constitute the final step in 
evaluation and should be the basis for arriving at some conclusions. 

The conclusions are very important in the evaluation of ESEA— Title I 
projects. The value of the entire project should be summarized to indicate 
the effect which the project has had on children, teachers, parents, and the 
community. 
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DIRECTIONS FOR REPORTING EVALUATION OF 
ESEA - TITLE I PROJECTS 

Enclosed are copies of the forms for reporting evaluation of ESEA- 
Title I projects. There are three of these forms which are identified 
as EV1, EV2, and EV3. Each has a different purpose and each is to be 
submitted at a different time. 

All reports on evaluation should be submitted in duplicate to the 
Office of Evaluation, Department of Public Instruction. If a project 
consists of several different instructional areas or services, a separate 
set of evaluation reports shall be prepared for each separate phase of 
the project. 

To determine the breakdown of a project for evaluation, reference 
should be made to Item 13 of Part II of the Project Application. Item 13 
required a description of each separate project activity or service. Also, 
it was necessary to prepare a sub-budget for each separate activity or 
service in the project as outlined in Item 13 • Separate evaluations should 
be made and reported for each phase of the project appearing in Item 13 and 
for which a sub-budget was prepared. 

ESEA-I-EV1 - Procedures to Evaluate ESEA Projects 

This form is to be prepared immediately following approval of the pro- 
ject. Duplicate copies of EV1 shall be forwarded to the Office of Evaluation 
prior to the inauguration of the project. 

The material contained in EV1 should follow closely the items and code 
numbers as they appear in the application form. It is essential that a 
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separate EV1 be prepared for each phase of the project. 

Copies of all EV1 forms should be retained by the LEA for reference in 
the subsequent preparation of EV2 and EV3. 

ESEA-I-EV2 - Evaluation Progress Report 

This report is to be submitted to the Office of Evaluation near the 
middle of the term covered by the project. For example, if the project covers 
four months, an EV2 Progress Report should be submitted at the end of two 
months. If the project runs for an entire school year, an EV2 should be sub- 
mitted at the end of the first semester. No EV2 report needs to be filed 
for short summer projects or other projects which are completed in four to 
six weeks. 

For the preparation of EV2, reference should be made to the information 
contained in EV1. However, EV2 should report any changes or modifications 
in objectives, instructional areas, services, techniques, or measurement of 
outcomes which result from necessity or experience in the conduct of the pro- 
ject. If, for example, EV1 indicated that personnel classified as special- 
ists will be employed, but no one with the specified qualifications was avail- 
able, EV2 should reflect this fact and indicate what is being done about it. 
Or, if certain materials or tests were specified on EV1 but these were not 
available for some reason or other, this fact should be reported on EV2 as 
well as the substitutions which have b*td to be made. 

If direct evidences of outcomes have been determined at the time of 
submitting EV2, these should be reported. Otherwise, evidence of achieving 
the objectives should be in terms of such subjective evaluations as condi- 
tions warrant at the’ time of submitting this report. 



Note again the necessity of submitting a separate report for each phase 
of an ESEA-Title I project. 

ESEA-I-EV3 - Summary Evaluation Report of ESEA Projects 
This report is to be submitted to the Office of Evaluation as soon as 
reasonably possible after the completion of the project. Reference should 
be made again to EV1 and EV2 as a basis for completing EV3« 

The value of direct evidence of outcomes is emphasized. Whenever pos- 
sible, objective measures should be employed and the results reported on EV3. 
Whenever standardized tests are used, the name and form of the test should 
be reported. 

Many devices for obtaining indirect evidence of outcomes are possible 
.and these should be used to supplement test results and other objective 
* measures. Indirect evidence will have to suffice where no direct evidence 
is possible. 



On-Site Evaluation of ESEA - Title I Projects 
A representative number of schools will be selected for on-site inspec- 
tion and evaluation of ESEA-Title I projects during the year by personnel 
from the Office of Evaluation. These on-site inspections will have no bearing 
on the submission of evaluation reports. 
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ESEA-I-EV3 Project Number Assigned by 

(Revised 7-66) Department of .Public Instruction 



COMMONWEALTH OF PENNSYLVANIA 
Department of Public Instruction 

Summary Evaluation Report of ESEA Projects 



PROJECT TITLE: 

i 

Code No. from item 11 of Project Application * — - 

School or agency submitting 

Address C° un, V 

Authorized LEA representative — 

# 

Phone, including Area Code 

Date submitted — 



1. To what degree were the Project Objectives 
(goals) met? 

(Please list your objectives) 



a. 



b. 



c. 



d. 



e. 



Please check 
Appropriate Column 
3- Excellent 2-Fair 1-Poor 



3 2 1 




(1-a) 



(1-b) 



(1-c) 



(1-d) 



(1-e) 



f. 



(1-f) 









.1 IUJ * W«UJI* J III * JPI I (Will ). JPPI.U !. ..'ll' M ! I'M 



wwm i '*i.i' w w vmvm 'V'"™mmmwi* l ******^i*^ 1 ********* 




; 

l 



g- 

h. 



i. 



(Please see pages 5 and 6 for definitions of terms) 



2. To what degree does the project provide for 
comprehensiveness? 

Comment: 
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(1-g) 

(1-h) 

(1-i) 



( 2 ) 
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3. To what degree does the project provide for 
balance? 

Comment: 




(3) 



i 



4. To what degree does the project involve 
coordination and continuity? 

Comment: 



(4) j 

I 



5. To what degree does the project provide for 
flexibility? 

Comment: 



(5) 



i 



6. To. what degree does the project draw upon 
specialists? 

Comment: 



( 6 ) 






7. To what degree does the project make use of 
education research? 

Comment: 



(7) 



8. To what degree does the project permit 
universality? 

Comment: 



( 8 ) 



9. To what degree does the project contribute to 
functional learning? 

Comment: 



(9) 
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10. To what degree does the physical facilities 
relate to the project? 

Comment: 




11. To what degree are the instructional materials 
utilized in the project? 

Comment: 



01 ) 



12. Direct (objective) Evidence of Outcomes 

(For Standardized Tests, indicate Name and Form of Test.) 
(Describe briefly locally devised tests.) 



13. Indirect (Subjective) Evidence of Outcomes 
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CONCLUSION 



I. Outstanding aspects of the project: 



II. Recommendations for improvement of the project: 



III. Comments: 



Please submit in duplicate to the Office of Evaluation, Department of Public Instruction, 
Harrisburg, Pennsylvania. 
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ESEA EVALUATION 
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Definitions of Evaluation Items 


Comprehensiveness: 


A project has comprehensiveness when it provides 
for consideration by the individual and group of 
the demands made by our complex society and 
assists these individuals and groups to more ade- 
quately meet these demands and satisfy their needs 
through growth as a result of this project or program. 


| Balance: 


A project has balance when proper emphasis is 
given to the various experiences and concepts so 
that the true objectives are achieved with minimal 
effort or distortion of values. 


Continuity: 

! 

i 


The program provides the opportunity for the pupils 
to grow progressively in knowledge, competence, and 
value appreciation. For example, adolescents are 
repeatedly provided with situations which help them 
build attitudes that characterize sound development. 
Such attitudes as self-discipline, tolerance, critical 
mindedness, social sensitivity, optimism, personal 
responsibility, self-respect, self-reliance and 
leadership. 


Flexibility: 


The program has been designed to permit the agency 
to revise or change the pattern of the program. 


Specialist: 


The preparation of the teacher or consultants in- 
volved in the program is such that as a result of 
education, training, and experience she can be 
considered an expert in the problem area. 


Research: 


The project was based and designed after consider- 
ation and review of surveys indicating local need 
and education research in the areas of individuals 
growth, learning and development as well as curriculum. 


Universialty: 


The term used to express the provision that enables 
the problem area to provide a relationship between 
various subject and problem areas. 


Functional Learning: 


The term is used to note the provisions of the pro- 
gram to provide the pupil with an opportunity to relate his 
studies to everyday life (living). 

» 
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Objectives (Goals): 



Direct Evidence: 



Indirect Evidence: 



ESEA-I-EV3 
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The term is used to present the statements of 
desirable educational attainment stated in 
operational terms. 



A term used to include such evaluation tech- 
niques as measuring student achievement, 
attitude, interests, study skills, social 
adjustment and reading readiness. 



A term used to include such evaluation tech- 
niques as faculty analysis and participation, 
self-evaluation, including a review of admini- 
stration practices, instructional material, and 
physical facilities. 
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